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Abstract 

We consider Backward Stochastic Differential Equations in a setting 
where noise is generated by a countable state, continuous time Markov 
chain, and the terminal value is prescribed at a stopping time. We show 
that, given sufficient integrability of the stopping time and a growth bound 
on the terminal value and BSDE driver, these equations admit unique 
solutions satisfying the same growth bound (up to multiplication by a 
constant). This holds without assuming that the driver is monotone in y, 
that is, our results do not require that the terminal value be discounted at 
some uniform rate. We show that the conditions are satisfied for hitting 
times of states of the chain, and hence present some novel applications of 
the theory of these BSDEs. 



1 Introduction 

Since their early introduction by Bismut [1| (in the linear case), and in partic- 
ular since the nonlinear existence result of Pardoux and Peng [H], Backward 
Stochastic Differential Equations have risen to be a powerful component of the 
stochastic analyst's toolkit. The early conditions of the theory, that the driver 
of the BSDE be Lipschitz, that the terminal time is deterministic, that the fil- 
tration is generated by a Brownian motion, have all been extended, leaving a 
rich class of equations which can be used in many practical problems. In par- 
ticular for our discussion here, the theory of BSDEs when noise is driven by a 
finite or countable state Markov chain (in continuous time) has been developed 
by the author and collaborators in a series of recent papers ([4], [7], [3], [8]). 

In this paper, we consider a novel extension of the theory of BSDEs, when 
the terminal time is replaced by an unbounded stopping time. We say this is 
a novel extension, however previous work has approached this question in the 
Brownian setting, in particular Darling and Pardoux [2], Briand and Hu [5] 
and Royer [21j . In those papers, a key assumption of monotonicity of the driver 
is made, essentially corresponding to discounting the future at a rate which is 
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bounded away from zero. This allows these authors to show that the problem 
is well-posed, in particular, that the BSDE admits unique solutions and that 
these solutions can be well approximated using finite-time BSDEs. A related 
setting is discussed in [5J Section 2] in the Markov chain case, again with an 
assumption of a discounting term. 

In this paper, we approach the problem somewhat differently. We seek to 
show that one can impose conditions on the stopping time directly such that 
the BSDE admits unique solutions (at least, unique solutions satisfying some 
integrability conditions), and that these conditions can be verified, for example, 
when the stopping time is a hitting time. In this case, our conditions can be 
connected to the uniform ergodicity of the Markov chain under a family of 
measures. These questions of ergodicity were explored in [5] in the context of 
proving the existence of Ergodic BSDE solutions, and we draw liberally on these 
results. In all our analysis, we do not assume the monotonicity of the driver, 
which in particular allows us to consider the case when the driver depends only 
on the Z component of the solution, that is, when the BSDE is effectively 
performing a nonlinear change of measure. 

We then present a couple of applications, both the standard application of 
BSDEs to control problems, and a novel application to our understanding of 
non-Ohmic electronic circuits, by relating these to a Markov chain model. 

The paper first presents the basic theory of BSDEs on Markov chains (Sec- 
tion [TTT|), then gives our existence proof (Section [5]), then considers how the 
conditions of the existence proof could be verified for hitting times (Section [3]), 
and ends with applications (Section 

1.1 Introducing BSDEs on Markov chains 

Consider a continuous-time countable-state process X in a probability space 
(rijj-", P). We shall suppose that X is a Markov chain in its own filtration, 
under a measure P. Without loss of generality, we shall represent X as taking 
values from the standard basis vectors Ci of (where e N U {oo} is the 
number of states, and denotes the space of infinite real sequences). We 
write X for this set of basis vectors. For notational simplicity, we will think of 
all vectors as column vectors, and denote by z* the transpose of z (so that z*y 
is the Euclidean or ^2 inner product, and e*z is the ith component of z). An 
element w G can be thought of as describing a path of the chain X. 

Let {-Ft}t>o be the completion of the filtration generated by X, that is, 

Tt = <yi{Xs}s<t) V {B e J-oo : P(B) = 0}. 

As A" is a right-continuous pure jump process which does not jump at time 0, 
this filtration is right-continuous, and we assume J- = Too — Vt<cxD -^t- ^'-'^ ^^e 
basic theory of continuous-time countable-state Markov chains, see for example 
Rogers and Williams |20l Vol. 1, p228ff] , for the approach taken here, see Elliott, 
Aggoun and Moore [111 Part III]). 

Let A denote the possibly infinite rate matri?!!] of the chain X. Note that 
{^t)ij > for i 7^ j and J2ii^t)ij — for all j (the columns of A all sum to 0). 

^In our notation, as in |14l . A is the matrix with entries Aij, where Aij is the rate of 
jumping from state j to state i. Depending on the convention used, this is either the rate 
matrix or its transpose. In our notation A* , the transpose of A, is the generator of the Markov 
chain. 
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We assume, for simplicity, that the entries in At are uniformly bounded, and so 
the chain is regular. 

From the Doob-Meyer decomposition (see [HI Appendix B]), we write our 
chain in the following way 



Xt = Xo+ AuXu-du + Mu (1) 
J]o,t] 

where M is a locally- finite- variation pure-jump P-martingale in R^, and the 
chain starts in state Xq € R^. Our aim is to study a class of BSDEs up to 
stopping times, that is, equations of the form 

Yt^i+ I f{uj,u,Yu-,Zu)du^ 1 ZldMu, 0<i<T<oo, (2) 

J]t,T] J]t,T] 

where r is an integrable stopping time, f : ft x R+ x R x R^ ^> R is a given 
function, y is a real-valued cadlag stochastic process, Z is a predictable process 
in R^ such that 

J]o,t] iJ]o.t] 

is a square-integrable martingale, (here (•)' denotes the ith component of the 
vector), and ^ is J>-measurable. The key problem is that we do not assume 
that T is uniformly bounded, nor that / has monotone dependence on Y. Such 
assumptions are needed to apply the 'discounted' BSDE methods in [5j or [2]. 

Remark 1. We note that the use of left limits for Y in the driver term of 
([2]) initially seems unconventional, for those used to the theory of BSDEs in a 
Brownian setting. However, it is the natural approach when the driver term can 
itself jump (see [5]), it ensures that the driver is predictable, and as the integral 
is with respect to Lebesgue measure and our processes have at most countably 
many jumps, in this case the equation is unchanged whether the left limits are 
included or not. 

Of importance will be the following process and the associated spaces. 
Definition 1. Let 

xPt dmg{AtXt^) - Atdmg{Xt-) - di&g{Xt-)Al. 

Then is a predictable process taking values in the symmetric, positive semidef- 
inite matrices in R^^^, with the property that 



Z*dM, 

o.t] 



E[Z*iJjuZu]du 
o,t] 



for any t and any predictable processes Z of correct dimension (see ^j). For 
simplicity, we write 

and note that this is a stochastic seminorm. 
We define the following spaces of processes. 

• Y ^ i/i?[suptgR+ Y^] < oo and Y is cadlag. 
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• Z E Hf j ^/^[ijo co] ll^tllMt'^^] '^^^ ^ predictable, 

• Z Z' if \\Zt — -^t'lU/t = for almost all t. 

• y G S'j^.ioc if {YtIt<T} G S"^ for any T < oo, and similarly Hfj j ^^^^. 

We note that the definition of S^ ^^^ is not the same as the set of processes locally 
in S'^ as usually understood in stochastic analysis, as the requirement is for any 
deterministic T, rather than for a sequence of stopping times. 

The basic existence theorem for BSDEs in this setting is the fohowing. 

Theorem 1. LetT be a finite deterministic time, and f : fix [0, T] xRxR^ M 
be a predictable function. If f is uniformly Lipschitz in y and z, that is, there 
exists a constant c > such that 

\f{u;,t,y,z)-f{Lo,t,y',z')\^<c{\y~y'\'' + \\z-z'\\liJ, 

and 

e\ I \f{uj,t,0,0)\^dt] < oo 

then for any ^ G L'^{J-t), there exists a unique solution (V, Z) £ S*^ x H^j to 
the BSDE 

^ = Yt~ [ f{Lu, u, Y^, Zu)du + [ ZldMu. 

J]t,T] J]t,T] 

Proof. For the finite state case, this result is given in For the infinite state 
case, we use the martingale representation result established in |4j, which natu- 
rally extends to general spaces, coupled with the existence result for BSDEs in 
general spaces established in [5]. □ 

We recall a key definition from [S]. 

Definition 2. Consider A and B (possibly infinite) rate matrices, that is, ma- 
trices with Aij > for i ^ j and X^i ~ /'''^ h '^'^d similarly for B. 
We write for the expectation under the measure where X is a Markov chain 
with rate matrix A. 

For 7 > 0, we shall say that B is 7 -controlled by A whenever B — jA is 
also a rate matrix, and the diagonal entries of B — 7 A are at most —7. If B is 
"/-controlled by A, then we shall write A B . 

If A :<j B and B :<-y A, we shall write A B. 

A key result in the analysis of BSDEs is the comparison theorem. In the 
case of BSDEs with Markov chain noise, and in general for BSDEs with jumps, 
a further condition is required to ensure that the result holds. In [71 [3] a general 
condition under which the comparison theorem holds is presented, and in [5] a 
condition specific to finite state Markov chain BSDEs was also given. We here 
give another variant, which in some sense underlies the others. 

Definition 3. For a driver f, we say that f is 7-balanced if there exists a 
random field X : D, x R+ x x ^ R^ , with A(-, -,2,2') predictable and 
X{uj,t, •, •) Borel measurable, such that 



/(w, t, y, z) - /(w, t, y, z') = (z - z')*{X{lu, t, z, z') - AXt^), 
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for each Ci € X , 

e*X{uj,t,z,z') 



e [7,7 '] 



for some 7 > 0, where 0/0 := 1, 

• l*X(Ld, t, z, z') = 0, for 1 G R"'^ the vector with all entries 1 and 

• A(a;, t,z -\- al, z') — X{uj, t, z, z') for all a £ R. 
For simplicity, we write X^''' for X{uj,t, z, z'). 

Remark 2. A minor variation on the proof of fSl, Lemma 12] shows that for / 
to be 7-balanced it is sufficient, but not necessary, that for all y, z, z', 

f{uj,t,y,z)~ f{uj,t,y,z') 1 

up to indistinguishability. A significant case where this condition may not hold 
is when f{uj,t,y,z) = z*{B — A)Xt-^ for some B A. Here we see directly 
that / is 7-balanced with Aj'^ — BXt-. 

Lemma 1. // / is "/-balanced, then it is Lipschitz with respect to z under the 
II • lU/t -seminorm. 

Proof. For each v G R^, we can write, 

MM,^v*i^tv= J2 {e*v.-X:_v)'{e*AXt-). 

Therefore, 

{v*AXt-)^ = - - X:_v){e*AXt-){e*AXt-) 

{i,j:ei,ej^Xt-} 

< Yl {e:v,~x:_v)^e:AXt-) Yl 

^\Xt-AXt-\\\v\\l,. 

(3) 

As we assumed \Xt-AXt^ \ is bounded, this shows that v ^ v* AXt- is Lipschitz 
in the || • [|j\/j seminorm. By assumption, we have 

f{u, t, y, z) ~ fiu:, t, y, z') = (z - z')*(A^''' - AXt-), 

so, as Aj'^ — DAXt^, for D some diagonal matrix with diagonal entries in 

[7, 7""'"], we know 

f{co,t,y,z)-f{uj,t,y,z') = {z - z'TiD - I){AXt^) = {{D - I){z- z')y{AXt^). 
Therefore, by ©, 

(/(w,i,y,z) - f{uj,t,y,z')f < \Xt^AXt-\ \\iD - /)(z - z')||L,. 
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As / is 7-balanced we know /(w, t, y, z) — f{uj, t, y, z + al) for any a, so without 
loss of generality we can write z — z' in a form such that X^_{z — z') = 0. For 
each Ci we have {D — T)ei ~ KiCi for some /ti € [7 — 1,7^^ — 1]. As 7 < 1, for 
these z-z' =: } ^^^e,, 

\\{D~I){z-z')\\l^^\\ J2 '^^^^^C/ " ^ {K^,v,ne*AXt-) 

{f.e.^Xt-} * {i-.e.^Xt-} 

E vUe*AX,.)^^-'\\z-zTM,, 

which yields the result. 

□ 

In the following lemma, we use the notion of the essential supremum of a 
family of random variables (as opposed to the essential supremum of a single 
random variable) . This is similar conceptually to the supremum taken pointwise 
for each w, with more care taken to ensure measurability and to prevent sets of 
measure zero contaminating the result. A construction and discussion of this 
concept can be found in [151 Appendix A5]. 

Lemma 2. Let {f{u;uj,t,y,z)}ueu &e a family of ^-balanced drivers. Then 
g{uj, t, y, z) := ess sup{/(w; w, t, y, z)} 

is also ^-balanced, and similarly for essinf„g[/ /. 

Proof. Suppose first that for almost all w, all t,y,z the supremum is attained. 
Then, omitting uj,t,y for notational simplicity, there exists some predictable 
control u* such that 

g{z) ~ 9{z) < fiu; z) - f{u*-z) = (z - -z){\* - AXt-) 

for some A* satisfying the requirements of Definition [3] for f{u*'. • • • )■ Similarly, 
there exists some m*. A* such that 

g{z) - g{z) > f{u^;z) - f{u^;z) = (z - z)(A* - AXt-) - e. 

As g(z) — g[z) is scalar, we see that for 

^ _ g(z) - g{z) - f{u^;z) + f{u^;z) 

f{u*;z) - f{u*;z) - f{u^;z) + f{u^;z) 

we have 

g{z) ~ g{z) = (z - z)(aA* + (1 - a)A* - AXt-). 

Finally, we note that A^'^ := aX* + (1 — a)A* satisifes all the requirements of 
Definition [31 

If the supremum is not attainable, then it is possible (but tedious) to con- 
struct an appropriate approximation sequence and to show that the correspond- 
ing vectors A have a convergent subsequence in £2 , using the boundedness prop- 
erties of A. The limit of this sequence will then satisfy the requirements of the 
theorem. The details are left to the reader. □ 
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Theorem 2 (Finite-time comparison theorem). Let {Y, Z) and {Y',Z') be the 
solutions to two BSDEs with drivers f and /'. Suppose f is j -balanced and 
f{uj, t, y, z) > f'{uj, t, y, z) for all (y, z), dt x dF-a.s. Then Yt >Y^ a.s. implies 
Yt > Y/ a.s. up to indistinguishability . 

The finite-time comparison theorem is easy to deduce (see for example [7]) 
from the following lemma. 

Lemma 3. /// is ^ -balanced, then for any predictable processes Z, Z' g t-ioc 
any process Y £ S'^y^^, any T < oo, there exists a probability measure Q"^ equiv- 
alent to P such that 

Mt^ f {fiLu,s,Ys^,Z,)- f{u:,s,Ys-,Zi))ds+ f {Z, - Z'^dM^ 

J]Q,tAT] J]0,tAT] 

is a Q'^ -martingale. 

Proof. Let Q-^ be the measure under which X jumps, at time t G [0, T], to a state 

Ci ^ Xf- at a rate e*Af'''^*, where A(- • • ) is the random field associated with / 
by Definition [3l This is a predictable bounded process (as A is bounded), and 
so the measure Q"^ is well defined. Under this measure, X has a semimartingale 
decomposition 

Xt^Xo+ [ du + Mt 



for Mt a ^ martingale. We can then verify that Mt — jjg (](-^ti — Zu-)*dMu, 

and a calculation of the predictable Q-^-quadratic variation of Mt guarantees 
the desired martingale property. As the relative rates of X's jumps under P and 
Q are bounded, by assumption on A, we can deduce that Q"'" and P are indeed 
equivalent measures. □ 

The following lemma extends this measure to all time horizons, and expresses 
its properties in a useful manner for the study of BSDEs. 

Lemma 4. Let Y and Y be two processes in S^ ^^^ with dynamics 

dYt = -/(w, t, Yt-,Zt)dt + Z*dMt 
dYt = -fiuj, t, Yt^ , Zt)dt + Z;dMt 

for some processes Z, Z ^ H\j j j^^ and some ^-balanced drivers f , f . Then 
there exists a measure Q such that 

Yt^Yt+ f {f{uo,s,Ys-,Z,)~ f{u:,s,Ys^,Z,))ds 
is a '[^-martingale. 

Proof. For each T, let be the measure given by Lemma |31 Then for t <T, 
a simple rearrangment shows that 

Yt-Yt-\- [ {fiu,s,Y,^,Z,)~ f{Lo,s,Z-,Z,))ds 



Y0-Y0+ {f{uj,s,Y,^,Z,)- f{u;,s,Y,_,Z,))ds+ {Z.^Z^fdM, 
' 1 hoA 
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and by Lemma [3] this is a Q-^-martingale up to time T. We now note that 
the measures are consistent, in that Q"^|j^t = Q*|j^t for any t <T. As our 
underlying space is the space of paths of a countable state Markov chain, which 
can be embedded in the space of paths in R, Kolmogorov's extension theorem 
implies the existence of a measure Q with Q\j^^ ~ Q'^Ij^'t fo'" ^- ^ 

Definition 4. Let Qj denote the family of all measures Q where X has compen- 
sator A(w, t), for A a predictable process with 1*X = and (e*A(w, t))/ {e*AXt-) G 
foT all i, where 0/0 :— 1. 

Remark 3. Note that P e Qj, the measures which appear in Lemma |4] are in 
Q-y, and, for any Q,Q' G Q^, any stopping time r, the measure defined by 
Q{A) = E^[E^' [lA\Tr]] is also in Q^. 

It is worth noting that, in general, the measures in Qj will be singular, 
even though their restrictions to J^t are absolutely continuous for every T. The 
following lemma gives us a slightly stronger result. 

Lemma 5. Let t be a stopping time with r < oo a.s., and let Q G Q^. Then 
Q|jr^ and are equivalent. 

Proof. As the relative rates of jumping are bounded, we know that V\j^^ and 
QIj^j, are equivalent for any T < oo. Let A G Jv. Then by definition of J-V, 

P(.4 n {r < T}) = ^ Q{A n {r < T}) = 

Now, as r is almost surely finite valued, by the monotone convergence theorem 

lim F{A n {t < T}) = F{A) 

T— f oo 

and similarly for Q. Hence 

P(^) = 4^ P(yl n {r < T}) = VT 

^Q{Ar\{T <T}) ^0 yT^Q{A)^0 

and so P and Q are equivalent on J>. □ 

2 BSDEs to Stopping times 

In the following, we make great use of the following version of Markov's inequal- 
ity. 'For any stopping time r, any T > 0, E[Lr>T\J^t] < Elr^+f^lTt] T-^^+i^^' 
Our key result is as follows. 

Theorem 3. Suppose ^ is -measurable and for some nondecreasing functions 
K, K : — > [1, oo[, some constants /3, /3 > 0, 

E'^imTt] < K{t), i?Q[(l + T)i+^|J-t] < K{t), E^iiKirY+^lTt] < K{t), 

all P-a.s. for all Q G Q-y and allt. Let f : n xR+ xRxR^ ^ R be -/-balanced, 
and such that for any y,y' , z, 

|/(c^,t,0,0)| <c(l + t'^), 
f{^,t,y,z) - f{uj,t,y',z) 

; e -c, 

y-y 
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for some c S K, some /3 € [0,/3[. Then the BSDE (0) admits a unique adapted 
solution satisfying the bound 

\Yt\ < {l+c)K{t). 

Proof. We first prove a solution exists. Let t„ :— t An. Define 

/ , X f{^.u,y,z)- f{Lo,t,y',z) 
r{uj, u,y,y , z) , (4) 

v-y 

and notice by assumption that r G [0, c] and is predictable. Define (F", Z") to 
be the solution to the finite-horizon BSDE 



J]t,n] J]t.n] 



(5) 



As ^ is 7> measurable, we see that Z" = whenever u > t. Now for n > m, 
by Lemma [3] and Ito's Lemma, there exists a measure Q (depending on n,m) 
such that 



fiLO,u,Y:\,0)du 



exp 



r(w,u,r"_,0,0)du 



/(w, u, 0, 0)du 



yn _ y ™ ^ 



- e/r<™) + / /(c., rjL ,Z2)-f{u;,u, , Z^) 

]«,r„] 



exp I 



]t.r„] 



-Ft 



Combining these, as r > and by our assumptions |(^| < K(t), we see 

\Yr-Yn<E^mr<n-iIr<rn\+ f \f{uj,U,0,0)\du\J^t]\ 

J]r,„,r„] 

< E^[{K{T))Ir>m + c(l + r'^)(r„ - r,„)|J-t] 

< i?Q[if(r)/,>„,|J-t] + ci?Q[r(l + t'^)/.>„,| J-*] 
<i?Q[if(T)i+^|J-t]i/(i+«i?Q[/,>„|J-t]^/(i+« 

+ ci5Q[r(l + r'^)|J-,]i/(i+")i?Q[/,>„| 



where a := ^—7-. Hence, writing a := (1 + B)( A tt— I , as we know a > 0, 
we have a bound of the form 

\Yr-Yn<K{t)m-'' 

for some nondecreasing function K{t), and so is a Cauchy sequence (con- 
verging uniformly in u on compacts in time). 



9 



Now define Yt :— limnFj". It is easy to see that this satisfies the desired 
dynamics. Furthermore, for some measure Q we have the representation 

Yt = E'i \e- ^I'-i '■(--.^"- '0.")'^" f f{uj,u,0, 0)du) Tt 

from which, given our assumptions on / and r, we can see that 

\Yt\ < il + c)K{t). 

Now suppose we have two solutions Y and Y. Then from the assumed bound 
and the fact Yt = Yr = £, on the set t < T, we can see that for some measure 
Q given by Lemma [31 

E^[\Yt -Yt\]^ E^[\E%-^i*-^--i r{^,u,Y^-,Y^-,Z,)du^y^ _ YT)\Ft]\] 

<E''[E'i[\YT~YT\\Tt]] 
<E'[2{l + c)E'^[K{T)Ir>T\Tt]] 
<2{l + c)E^[E'^-[K{T)Ir>T\m 

< 2(1 + c)i?'[£;Q[if (t)1+^| £;'[i?Q[/,>T|J-t]]^/(^+^) 
,2(i..M0)Va-)(3)^^^^^^^ 

< 2(1 + c)i^(o)i/(i+«i^(t)^/(i+«r-^. 

As ;3 > 0, letting T oo implies that E^[\Yt -Yt\] =0 for ah t, and so the 
solutions coincide almost surely. As the solutions are cadlag, they agree up to 
indistinguishability. □ 

Remark 4. We emphasise that the above proof permits the case when / does 
not depend on y. In fact, the requirement that / is monotone decreasing is not 
strictly necessary for our proof. It is enough to guarantee that the discounting 
terms Jj^ r(LLi, u, y, y' , z)du are uniformly bounded above for all s < t and all 
processes y, y' , z, as this just introduces a positive constant into our estimates, 
which will then also appear in the bound on Yt. 

We now extend our existence result to drivers which are only 'locally' 7- 
balanced and Lipschitz (in y), under slightly more restrictive assumptions on 
the terminal condition and the driver evaluated at zero. 

Theorem 4. Suppose ^ is J- ^ -measurable and for some constant k 
\^\<k, \f{u;,t,0,0)\<k, E'^[{T-t)+\Tt]<k 
and for some /3 > 0, some nondecreasing functions K, K :M ^ [^,oo[ 
E%1 + r)i+^| J-t] < K{t), E^^lKiry+^lJ^t] < Kit) 

all P-a.s. for all Q G and all t. 

For y,n E M, let y^"' := (—71) V y An, and for z G let z*^") be the vector 
with components {e*zY^^K For f : predictable function, 

suppose that 

/(")(c., t, y, z) /(^, t, [z ~ (X;_z)l)(")) 
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satisfies the requirements of Theorem \^ for every n > 0. (That is, Z*^"^ is 7- 
halanced in z and Lipschitz decreasing in y, where 7 and the Lipschitz constant 
can depend on n). Then the BSDE (0) with driver f admits a unique bounded 
solution. 

Proof. First note that as /^"■' satisfies the requirements of Theorem [3] for any 
n, the BSDE with driver /^"^ admits a unique sohition We know that 

this solution has a representation 

y/' = £;Qre-^i*-i"^"'"'''"-°'°'''"(e+ / f^'^\uj,u,0,0)du) J"/ 
L ^ J]t,r] ' J 

for some Q G Q-y", where r is as in (jH), and so 

/ |/("H^,",0,0)|dw J-J < fc + fc£;^[(r-t)+|J't] < fc(l + fc). 



\yt\ < 

From the fact that AF/* = (Z")*AXf and AXf = — for some i,j, and that 
jumps of X are totaUy inaccessible and Z is predictable, we see that \e*Z"\ < 
2fc(l + fc) up to equivalence ^m- Hence we have a bound on F", Z" independent 
of n. Therefore, provided n > 2fc(l + k), we see that Y^\Z^ is a solution for 
the BSDE ([2]) with driver /, as the truncation will have no effect. 

Conversely, suppose we had two bounded solutions (F, Z), (F', Z'). Then we 
could set n > k' , where k' is a bound on the solutions, so that the truncation has 
no effect, and both processes would solve the BSDE with driver /*■"•*. However 
this BSDE satisfies Theorem [3] and so admits unique solutions, hence we have 
a contradiction. □ 

Remark 5. The peculiar definition of /("^ in the previous theorem (in particular 
the unnatural use of {z — {Xf_z)lY"'> rather than z^'^\ is simply to ensure that 
is Lipschitz continuous in the || ■ \\Mt seminorm whenever / is, as the basic 
truncation z t-^ z*^"^ is not invariant under || • equivalence. 

The conditions of these statements may seem unusual and restrictive, how- 
ever the following lemma gives a key example when they are satisfied. It is 
an immediate consequence of Corollary [T] in the next section, and so is stated 
without proof. 

Lemma 6. Let t be the first hitting time of a set 'E, C X. Let ^ be a random 
variable of the form 

for some function g with g{t, x) < k{l + 1^) for some k, (3 > 0. Then there exist 
functions K, K satisfying the requirements of Theorem O 

Remark 6. Theorem[3](but not TheoremU]) would work equally well for Brownian- 
Motion-Based BSDEs, with the corresponding definition of the family of mea- 
sures Q-y. 

Theorem 5 (Comparison theorem). Let {Y, Z) and {Y',Z') be the solutions to 
two BSDEs with drivers /, /' and terminal values f , ^' satisfying the conditions 
of Theorem\^ or^ Suppose f{uj, t, y, z) > f'{uj, t, y, z) for all (y, z), dt x dP-a.s. 
Then S, > C implies Yt > Y/ a.s. up to indistinguishability. Furthermore, 
Yt — Yl a.s. on A ^ Tt if and only if Ys = Y^ a.s. on A for each s > t and 
/(w,s,y,_,Z,) = /'(w,s,y;_,Z^) onAx]t,T], dFxdt-a.s. 
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Proof. From Lemma HI we know that there is a measure Q such that for any 
stoppmg time r* < r, 



Yt - y; = [Yr* - y;, + / f{uj, u, z„) ~ /'(w, u, Y:^^,z^)du 



]t,r'] 



From Lemma O we know that the stated assumptions ^ > C / > /' hold both 
P and Q-a.s. Suppose we have a set i? e Tt such that Yt — < on B. Then 
define the stopping time 

T* inf{s > t : - i;' > 0} < r. 

From this and the definition of r in it is easy to deduce that 



iB\yi-Yi\<E^ 



r{uj,u,Yu-,Yi_,Z^)lB\Yu- - Yi_\du 



However, as r > 0, the right hand side is nonpositive, so Ib^i ^ ^/I = 0, that 
is, Yt > Yt except on a null set. 

Now, for any A G J-"t, it is easy to verify from the representation 



Yt - - 



exp 



]t.r] 



r{Lj,u,Yu^,Y:_,ZMi^~e) 



Tt 



that Ia^i ^Y[) = if and only if Ia^s ^^i) — fo'" each s > t and thence 
j[u:,s,Ys-,Z,) = /'(a;,s,y;_,Z^) on Ax]t,r], dP x dt-a.s. □ 

For completeness, we now state a useful result which gives us a solution of 
the form Yt = uit^Xt). In [S] Corollary 2] this result is stated, but only for the 
case N < oo and T deterministic. 

Theorem 6. Let t be the first hitting time of a set S, and let Y he the solution 
to a BSDE with Markovian terminal condition Y^ — (/)(t, X^-), for some bounded 
deterministic function (j) ; xR^ — > R. Suppose f satisfies the requirements of 
Theorem\^ or^ and is Markovian in the sense that f{uj, t, y, z) — f{Xt-., t, y, z) 
for some f . Then there exists a measurablefunction u : x A" — >■ M such that 
Yt — u{t,Xt) for all t <t. Furthermore, u satisfies 

• u(t, x) = (f)(t, x) on R+ X S, 

• the associated vector Ut defined by e*Ut = u(t, Ci) satisfies the N -dimensional 
ODE system 

dut = -{f{t,ut)+ A*ut)dt fort<T 
where e*f(i, u) := /(e^, t, e*u, u), 

• and the solution process Z is given by Zt = Ut, up to equivalence ~a/- In 
particular, note that the BSDE has a solution Z which is deterministic. 

Proof. First note that, by the Markov property, we have 



Yt^E 



E 



Xr)+ f{Xu-,U,Yu-^,Zu)du 



T,Xr)+ f{Xu-,U,Yu^,Zu)du 



Tt 
Xt 



u{t,Xt 
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for some function u, where the third hne follows from the Doob-Dynkin Lemma, 
which implies that for each t, as Yt is (T(Xt)-measurable, Yt is equal to a mea- 
surable function of Xt. Right-continuity of y in t ensures that the function u is 
measurable in the product space. Clearly u{t, x) = cj){t, x) on the set t = t, that 
is, on R+ X S. Applying O Theorem 3.2] (modified trivially to allow = oo if 
necessary) we have the desired dynamics of u and the statement Zt — Ut- □ 

We can also obtain a version of this result under which Y does not depend 
on time. 

Theorem 7. Let t be the first hitting time of a set S C A" and let f : X x M+ x 
M.^ — > R satisfy the conditions of Theorem^ Consider the BSDE 



Yt = (l>{Xr) 



/ f{Xu-,Yu-,Zu)du+ / 

Zu 

J]t,T] J]t,T] 



where (f> is a bounded function A" — > R. Then there exists a hounded function 
u : X ^M. such that Yt = u{Xt) and e*Zt = u{ei), so Z is constant. The vector 
u defined by e*u = u{ei) satisfies the equation 

f{x, x*u, u) = —u*Ax for x E X \ E, 

with boundary values u{x) = 4>{x) for x £ S. 

Proof. As T is a hitting time of a Markov chain A, wc know 



Yt^E 



E 



4>{Xr) - 



f{Xu-,Yu-,Zu)du 



t.r] 



f{Xu-,Yu-,Zu)du 



Tt 
Xt 



Defining a new Markov chain Xg = X^-t for s > t, with hitting time t — t — t, 
by the Markov property we have 



Yt^E 



0,f] 



f{Xu-,Yu-,Zu)du 



and as the right hand side does not depend on t, we see that Yt is a function 
purely of Ao = At. 

Let A be the modification of the rate matrix such that Ax — Ax for a; ^ 5 
and Ax = for x G S. This agrees with A on the set < < r, so from Theorem 
m we see that there exists a vector u such that Yt — X^_u, Zt = u, and 

= -(f (u) + A*u)dt for t < T. 

However this equation does not depend on t, and so 

f(u)+i*u = 0. 

Premultiplying hy x € X and using the relation between A and A and the 
boundary conditions, we obtain the desired equation. Boundedness of u is trivial 
from this representation, as we have assumed /, are bounded. □ 
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3 Exponential hitting time bounds 



We now seek to show that when r is the hitting time of a state of the chain, the 
required r-integrabihty assumptions of Theorem |3] are satisfied. This is done by 
examining the exponential ergodicity of the Markov chain under perturbations 
of the rate matrix. 

A key result of ^ is that, when the driver / is 7-balanced, does not depend 
on time or Y, and depends on uj only through Xt{ui), the measures Q and P 
established by Lemma S] have closely related ergodic properties. In particular, 
if the underlying Markov chain is uniformly ergodic under P, then it is also uni- 
formly ergodic under Q, and the rate of convergence to the Ergodic distribution 
can be uniformly bounded for every Q in terms of 7 and the properties of P. 

This result is of fundamental importance to our approach to BSDEs up 
to stopping times. Our first step is to generalise away from the Markovian 
assumptions on /, and to work only with the assumption that / is 7-balanced. In 
this case, notions of 'ergodicity' cease to have a clear meaning, as the dynamics 
of the Markov chain under Q can be time-dependent. Instead, we work with 
a more fundamental property, that of the existence of exponential moments of 
the first hitting times of states of the Markov chain. 

We begin with the definition of uniform ergodicity. 

Definition 5. Let Ai denote the set of probability measures on X , with the 
topology inherited from considering them as a convex subset of li{X) (the total 
variation topology, with norm ||/||Ty = 'Ylix \fi^)\)- write P^fJ. for the law 
of Xt given Xq ~ when X evolves following the rate matrix A. 

We say the Markov chain X is uniformly ergodic if there exists a measure 
TT on X , and constants R, p > 0, such that 

sup ||P/^^ - tt\\tv < Re~^* for all t. 

In this case tt is the unique invariant measure for the chain. 

The following lemma is simply a variant of [18^ Theorem 16.2.2(iv)] and is 
stated without proof, but shall be useful in our understanding of this property. 

Lemma 7. Let X be a uniformly ergodic Markov chain, and let xc be an 
arbitrary state. Let tq be the first hitting time of xc ■ Then for some /? > 
(and hence for all (3 sufficiently small), supj.g;^' £'[e^'^'^ |Xo ^ x] < 00. 

We make the following basic assumption about our processes. 

Assumption 1. Under the measure P, the Markov chain X has time-homogenous 
rate matrix A and is uniformly ergodic. 

Theorem 8. Let X, X be two independent copies of the Markov chain on X , 
and let these be uniformly ergodic under the measure generated by a rate matrix 
A. For some arbitrarily chosen state x, let f = inf{i : Xt — Xt — x}, the first 
time X and X meet in the state x. Then for any e > 0, there exists /3 > 
depending only on 7 and A such that 

sup sup [e^^^lXo ^ x,Xo = x] < 1 + €. 
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Proof. This is a re-expression of [SI Lemma 11, Corollary 12], where we note that 
the T referred to here is not as stated in [5], but is rather the more restrictive 
stopping time used in the proof of [SI Lemma 11]. □ 

We can extend this result to give the following theorem. 

Theorem 9. For some arbitrarily chosen state x, let f — ird{t : Xt — x\ . 
Then for any e > there exists a /3 > depending only on A and 7 such that 
for any measure Q G Q~j, 

E%P^--')^ \Tt\ < (1 + e) 

for any t. 

Proof. Define the BSDE driver 

g~^{x,z):= sup {z*{B ~ A)x). 

By Lemma [5] and Remark [21 we can see that is 7-balanced. By Lemma [TJ g^ 
is Lipschitz with respect to the || • seminorm. 

Let f{oj,t,z) = z*{\t — AXt~), where A is a vector process generating Q G 
Q^, in the sense of Definition |4l For any T > we can consider the BSDEs 
with terminal value e'^''^^"^-' at time T, and drivers g^ and /. Let and 
denote the corresponding solutions, for which direct calculation shows that 

sup i?^[e^(^^^)|J-t] 

We can see that / is 7-balanced, and furthermore, that g^{Xt-, z) > fiuj, t, z). 
By the finite-time comparison theorem (Theorem [2]) we see that F/ > v/ for 
all t. Hence 

sup E^[eP^^^^^\Tt]>E%P^*''^^\Tt] 

B~.,A 

and taking T — > 00, as f is almost surely finite, by the monotone convergence 
theorem we see 

E%f^^\Tt] < sup E^[e^^\Tt]. 
Now under each rate matrix _B, 

E''[e^^\Tt\=E''[e^^'^^+^^-'^*^\Tt] < e^^'""^^ E'^le^^^-'^^ \Ft] 
For every a; G A", by the Markov property and Theorem |8l 

^sjg/3(f-t)+|^^ ^ ^] ^E^[e'^^\Xn = x] < 1 + e. 

Hence 

E%^^\Ft] = E'^[ef'^\Xt] < e'^(*^^)(l + e) 
and rearrangement yields the result. □ 
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Corollary 1. Let f he the first hitting time of a state x. Then for any /? > 0, 

any Q G Q-y, there exists a constant k such that 

E'^[[l + ff+^\Ft]<k{l + tf+^ 

Proof. From Theoreni[5]we know that for all 7 sufficiently small, E'^[e''^'^^'^^^ \Tt\ < 
(1 + e). Hence, we can see that for any /3 > there exists some constant k such 
that 

E'i[{l + {T-t)+Y+f'\Ft]<k. 
As i > 0, we have the bound 

< {\ + tf+^{i + {T -t)+Y+p. 

The result follows. □ 



4 Applications 

We now present some novel applications of these methods. We begin with the 
archetypal Markovian control problem. A related approach for ergodic control 
problems was considered in [S] . A general setting for control of a Marked Point 
Process to deterministic times is given in [TO]. 



4.1 Control to a stopping time 

Consider the problem of minimizing a cost 



Yt = essinfi;" 

u 

where 



L(S, Ys- , Xs- ,Us)ds + (t>{T, Xr) 
]t,r] 



• [/ is a space of controls, which is a separable metric space, 

• u is a J7 valued predictable process, 

• is a terminal cost function with a;) < c(l + 1^^ for some /3 > 0, 

• L : M+ xRxA'xf/^Misa measurable cost function, with 

\L(t,y,x,u)\ < c(l + t^), iL{t,y,x,u)-L{t,y',x,u))/iy-y') £ [-c, 0] 

for some c e M+, some /3 < /?, 

• E" is the expectation under which at time t, for the path w, X jumps 
from state to state ej at a rate e*A"*("^ei, for some measurable matrix 
valued function A*- ' : U — s- rate matrices. 



• for some 7 > 0, for all u £ U, the matrices A" A, for some reference 
rate matrix A under which AT is a uniformly ergodic Markov chain. 
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• T is the first hitting time of a collection of points in X. 

We shall write E for the expectation under which X has rate matrix A. 
We define the Hamiltonian 

fix, t, y, z) = inf y, x, v) + z*{A^ ~ A)x}. (6) 

As A" ~^ A, by Lemma [2] we see that / is 7-balanced and the requirements 
of Theorem |3] are satisfied. Therefore, the BSDE ^ with driver / admits a 
unique solution with bounded growth. By Theorem[Bl the solution to the BSDE 
is Markovian, that is, Yt = u{t,Xt) and e*Zt = u{t,ei), for some function u. 

If the infimum in (jG]) is attained, then there exists (assuming the continuum 
hypothesis, by McShane and Warfield FT7|) a measurable function k : X x R+ x 
RxR^ such that 

fix, t, y, z) = Lit, y, x, t, y, z)) + - A)x. 

We then have the following theorem. 

Theorem 10. In the setting described above, let iY, Z) be the solution to the 
BSDE (0j with driver f . Then the following hold. 



(i) For an arbitrary control u, if 



Y^ ^El\l Lis, + 0(r, X^) 



]t,r] 

then Y" > Yt , and equality holds if and only if 

Lit,Yt-,Xt-,ut)+iZt)*{A"*~A)Xt- = fiXt-,t,Yt-,Zt) dPxdt-a.e. 



(ii) If the infimum is attained in then the control — niXt- ,t,Yt- , Zt) 
verifies Y^ — Yt, and is an optimal feedback control. 

Proof. We see that Y-j- = Y^ = 0(r, Xt-). By definition of the Hamiltonian, 
fix, t, y, z) < Lit, y, x, Ut) + z*iA^ ~ A)x, and so by the Comparison theorem 
(Theorem [S]), we see Y^^ > Yt for every t, and the desired condition for equality 
holds. Statement (ii) also holds by an application of the Comparison theorem. 

□ 



4.2 Stochastic shortest paths 

Let Q he a directed graph, represented by nodes in X. Let the distance from node 
Ci to Cj be given by e*Dei for some matrix D. Let A be the transition matrix 

of a continuous time random walk on this graph, so that e*Aei = v^^^f/'^'^n^ -i 

for i ^ j, and e*Aei = — ^'j^^i- The time to reach a node x starting from 
a node x' is then given by £'[t|Xo = x'], where t is the first hitting time of x. 

Now suppose that it is possible to choose to 'walk faster' on those paths 
which are heading in the right direction. This could be modelled by a change of 
measure, where the transition matrix A is replaced by a matrix A^, where u is 
some control. Provided A" A for some 7, as shown in the previous section, 
the expected time to hitting x under the optimal control is then the solution 
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of the BSDE to r with driver /(x, z) — inf„{z*(y4" — A)x} and terminal value 

Alternatively, we can consider the expected remaining time to hitting x 
under the optimal control as the solution of the BSDE to r with driver f'{x, z) — 
inf„{z*(A" — A)x} + 1, and terminal value ^' — 0. If the earlier BSDE has 
solution {Y,Z), and this variant has solution {Y',Z'), then we can see that 
Yt ~Y^+t, but that satisfies the requirements of Theorem [71 and therefore 
its solution is of the form F/ = u{Xt) for some u : X M., which may be 
convenient for calculation. 

4.3 Reliability for networks with control 

Consider a model for transmission of messages over a finite network. A message 
is to be transmitted from a node a;o to a node xi, and each node Ci naturally 
passes messages to node ej at a rate e*Aei. We wish to examine the probability 
that the message eventually reaches its destination. When the network is flaw- 
less this probability is always one. However, suppose that each node x loses the 
message at a rate r^, and that there is a (possibly empty) collection of nodes 2 
at which a message is irretrievably lost. Without any control, we can consider 
this as a linear BSDE to a stopping time. 

Consider the Markov chain describing the motion of a message. Let r be the 
first hitting time of {a^i} U S, and let Yt be the solution to the BSDE 

I{x^=xi}^Yt- / -rxt.Yu-du+ / Z*dMu 

J]t,T] J]t,T] 

SO that 

Yt^Eie-^^-^^^-^^-'^I^x.^^^ylTt]. 

Now suppose we can extend this model, so that each node has some control 
over where a message is sent. Each node has a control u with which it can 
modify transitions so that they occur at a rate e*A^ej for some A" A, so as 
to maximise the probability that the message reaches its target. The maximal 
probability is then given by solving the BSDE 

I{x^=.^} =Yt- f -rx^-Ys- + sup{Z:(A" - A)X,_}ds + f Z^dM^, 

and the optimal policy for the active node Xt- at time t is given by 
u* = argmax{Zt*(A" - A)Xt-}. 

u 

4.4 Non-Ohmic Electronic circuits 

We now give a different situation, where we consider an electronic circuit. The 
theory of circuits of resistors is described using Kirchoff's laws and Ohm's law, 
a more detailed presentation of the approach we take here can be found in fTSJ 
Chapter 1]. 

Consider a circuit of resistors. Let the circuit be described by a graph 
represented in A", and let the edge {ei,ej) have resistance Rij- Then it follows 
from Ohm's law and Kirchoff's laws that, on a source set S, the voltage potential 
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is a prescribed function (j), and off the source set the voltage potential v in 
the circuit is a harmonic function. In particular, if m^j — l/Ri.j denotes the 
conductance over the edge {ei,ej), we have 

v{e.i) = (/)(ei), e S, 

Let A be the matrix defined by e*Aei = Wij for i j and e*Aei = — Ej '"'ij - 
Then we consider a Markov chain X with rate matrix ^, and can show that 

v{ei) ^ E^[4>{Xr)\XQ ^ e,] 

where r is the first hitting time of the source set. 

Now suppose that our circuit no longer consists purely of resistors, but that 
it also has diodes. These ubiquitous electronic components fail to satisfy Ohm's 
law, and so the above representation in terms of a Markov chain fails. However, 
it is possible to write down a nonlinear relationship between voltage, current and 
resistance which is satisfied (see for example, [11,, Section 5.2]). In particular, 
if we know the voltage drop over an edge Vi,j = v{ei) — v{ej) we can write 
Ii,jRi,j{Vi,j) = Vij, where is the current passing over the edge. From [TT] . 
the relation for an np-type diode, for example, is given by 

= r(exp(K,/y^)-i) 

where PjV^ are constants based on the properties of the diode. Rearranging 
this gives the implied resistance 

''-^''-^ = /^(exp(T/!lV)-l)>° 

which is a Lipschitz function of Vij, and is bounded away from zero (and oo) 
over any finite interval. 

Writing v for the vector with entries e*v — v{ei), and Wij{v) = {Rij{Vij))~^ 
for the implied conductance, we can construct a matrix A^ with e*A'^ei = Wij 
for i ^ j and e*A^ei = — Ej ^ij. For a given voltage potential vector v, this 

is the transition matrix based on the implied conductances at the potential 

V. 

Let A be as before, now defined by replacing each diode with a resistor with 
resistances Rij. As the circuit is finite, A generates a uniformly ergodic Markov 
chain. 

We then consider the BSDE 

q>{Xr) = Yt- [ Zl{A^' ~ A)Xs-ds + [ ZtdMt 

J%r] J%t] 

This is a time homogenous Markovian BSDE to a first hitting time. The ter- 
minal value (/>(^t) is bounded as for each n > 0, we can find 7„ > such 
that A"^ A whenever ||w|| < n, we see that the driver f{Xs^,Zs) = 

Z*{A^^ - A)Xs- is locally 7-balanced, and /(a;,0) = 0. By Theorem H this 
BSDE admits a unique bounded solution, and by Theorem [7] it is of the form 
Yt — v{Xt), which is precisely the voltage potential on the circuit with a diode. 

Using the same methodology, one can also extend this method to other 
electronic components, for example, transistors. 
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5 Conclusions 



We have considered BSDEs on Markov chains when the terminal time is replaced 
by a non-bounded stopping time, and no strict montonicity property in Y (i.e. a 
'discounting' term) is given for /. We have shown that, given appropriate poly- 
nomial growth bounds on the terminal value and the driver, and given sufficient 
integrability of the stopping time, these equations admit unique solutions with 
uniformly controlled growth in time. For the special case of first hitting times 
of a set, we have seen that the integrability conditions on the stopping time are 
indeed satisifed, and so these BSDEs admit unique solutions. 

We have then considered various applications to these equations, in partic- 
ular to control problems which terminate at a hitting time, and to problems of 
nonlinear behaviour on certain graphs, for example to network reliability. We 
have also seen how this gives a stochastic approach to our understanding of 
electronic circuits without Ohm's law. 

We can also extend these approaches to more general types of BSDEs, either 
to the classic Brownian setting of [19] , or to allow for Poisson jumps as in [22] ■ 
For the Brownian case, the required ergodicity properties may be derivable 
from [13) . however such an extension is non-trivial. On the other hand, if r 
is restricted to be the first exit time of a set, then standard estimates may be 
available to give the required integrability, in which case our results will extend 
directly. 
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